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The present invention relates to a method of and an apparatus for retrieving images and a method of and 
an apparatus for managing images, capable of efficiently retrieving desired dynamic images from dynamic 
images generated in a short period of time. 

As a method of retrieving desired dynamic images from those dynamic images which have been stored 
5 in advance, there have been a method for retrieving the desired dynamic images according to classification 
codes and associated information such as positional data of all dynamic images which are stored for each title 
of dynamic images and a method for visually retrieving the dynamic images by a user while actually feeding 
the dynamic images at a high speed by the user or directly retrieving the dynamic images at a specified interval 
while observing frame sequence. 
10 As a method for retrieving the dynamic images by using the images to be retrieved, one frame of images 
to be retrieved has been entered and a frame corresponding to the frame image entered has been selected 
from all frames for actual retrieval. 

However, in a method of retrieving the dynamic images according to the above-described classification 
code such as positional data, the user should store the positional data in connection with each title of the dy- 
15 namic image; for example, even though the user has wanted to know the data at a position on a video tape 
where the desired title is stored, the dynamic image could not be retrieved if there were no clue except the 
video tape. 

In a method of visually retrieving the dynamic images in rapid feeding, there has been a problem that, 
though this method can be executed if the number of dynamic images stored is small, a time required for re- 
20 trieval would be longer and therefore the method would be unpractical if a number of dynamic images are stor- 
ed. 

In a method of retrieving dynamic images by using images to be retrieved, it is necessary to determine 
whether a frame image entered identical to a frame image of dynamic images. 

Though pattern matching is available as a practical method of retrieval, there has been a problem that 

25 pattern matching for retrieval requires a lot of time. 

As the other method, a histogram of images and characteristics amount matching for determining whether 
the image includes special information (for example, character A) are available for retrieval. However, it is nec- 
essary to calculate in advance the characteristics amount for each frame as a whole at the time storage of 
images and a lot of time is required for such calculation. 

30 In other words, in case of the retrieval by using the characteristics of the image, for example, an attribute 
value information as "character "A" is included in the image", such attribute value information should be given 
to the storage process and the information should be manually entered to ensure accurate and minute entry 
of information and therefore a lot of time has been required and it has been practically impossible to carry out 
such input operation, depending on the number of subject frames. Lately, those studies as to automatic ex- 

35 traction of the attribute value information in accordance with the contents of images have been conducted ("Im- 
age Retrieval Adapted for Subjective Similarity" the treatise magazine of the Information Processing Society, 
Vol. 31 , No. 2, pp.227 - 236 and others). The information extracted from images has been controlled and stored 
as the array values at the characteristics amount level. However, the above-described automatic extraction 
also includes a problem in the aspect of accuracy. 

40 In this case, therefore, a difference if level exists between the attribute value information and the char- 
acteristics amount information to be provided for retrieval and another art to compensate this difference is re- 
quired for obtaining a proper accuracy. 

An object of the present invention is to provide a method of and an apparatus for retrieving dynamic images 
and a method of and an apparatus for managing images capable of solving one or more of the above problems. 

45 Another object of the present invention is to provide a method of and an apparatus for retrieving dynamic 
images and a method of and an apparatus for managing images capable of efficiently retrieving dynamic im- 
ages in a short period of time. 

One aspect of the present invention provides a method of and an apparatus for retrieving dynamic images 
by which a dynamic image is used as a search key. 

so A particular embodiment of the present invention provides a method of retrieving dynamic images corre- 
sponding to dynamic images entered for retrieval from dynamic images of a database, comprising a first ex- 
traction step for extracting a characteristics amount in accordance with a scene change of a dynamic image 
of the database; a second extraction step for extracting a characteristics amount in accordance with a scene 
change from the dynamic image entered for retrieval; and a determination step for determining a dynamic im- 

55 age corresponding to the dynamic image entered for retrieval, which is included in the dynamic images of the 
database, from said characteristics amounts extracted by the first extraction step and said second extraction 
step. 

Other aspects of the present invention aim to provide a method and an apparatus for retrieving dynamic 
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images and a method of and an apparatus for managing images capable of retrieving in a short period of time 
and saving manpower wherever possible. 

Yet other aspects of the present invention aim to provide a method and an apparatus for retrieving dynamic 
images and a method of and an apparatus for managing images which have novel functions. 
5 Other objects of the present invention will be obvious from the following embodiments and the accompa- 

nying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

10 Fig. 1 is a flow chart showing the steps of a first embodiment of the present invention; 

Fig. 2 is a block diagram showing a structure of hardware for implementing the present embodiment; 

Fig. 3 is an illustration of the scene change detection algorithm; 

Fig. 4 is an illustration of the scene change detection algorithm; 

Fig. 5 is an illustration of the scene change detection algorithm; 
15 Fig. 6 is a representation of the information to be added for the scene change; 

Fig. 7 is a diagram showing relational information of storage regarding the dynamic images entered; 

Fig. 8 is a diagram showing a step for extracting a characteristics amount sequence which serves as a 

search key; 

Fig. 9 is a diagram showing a step for determining a correspondence between the dynamic image entered 
20 and the dynamic image stored; 

Fig. 10 is a block diagram showing a second embodiment of the present invention; 

Fig. 11 is a flow chart illustrating respective modules built in the main storage 101 shown in Fig. 1; 

Fig. 12 is an example illustration of the designated region; and 

Fig. 13 is an example of the table. 

25 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Preferred embodiments of the present invention are described in detail, referring to the accompanying 
drawings. 

30 Referring to Fig. 1 , there is shown a flow chart of procedures of an embodiment according to the present 
invention. Processing steps can be classified into an input storage step for storing a plurality of dynamic images 
to be retrieved and a retrieving step for executing retrieval by using the dynamic images given to be retrieved. 
This embodiment is characterized with scene change automatic detection means 10 and 16, characteristics 
amount calculating means 12 and 18, and correspondence determination means 20. 

35 Referring to Fig. 2, there is shown a block diagram illustrating a structure of a hardware for implementing 
the present embodiment. Reference numerals 30 denote a CPU, 32 is a main memory which has the programs, 
data and CPU work area for executing the following processing, 34 is a video board for converting analog video 
signals from a video apparatus 36 to digital signals and transferring these signals to a frame memory 38 
through a built-in processor unit, and 40 is a console for displaying the image. 42 is a hard disk device and 44 

AO is a bus for transferring data between respective components of the hardware. 

The CPU 30 can directly access to the frame memory 38 and can therefore directly process digital dynamic 
images. 

In the present embodiment, scene information extracted and relational information given at the time of en- 
try and storage of dynamic images for preparing the database are stored in the hard disk 42, and the contents 
45 are referred for retrieval. 

In the present embodiment, scene change automatic detection means 10, 16 (Fig. 1) is provided both in 
an input storage circuit for dynamic images at the time of preparation of the database and in a retrieving circuit 
which applies dynamic images as keys. The scene change refers to a discontinuous change of the image se- 
quence as dynamic images such as, for example, changeover of the camera. In this embodiment, the scene 
so change is detected by actuating the scene change automatic detection means 10, 16 according to a software 
to be processed by the CPU 30. 

Dynamic images from the video apparatus 36 which are AID converted through a video input board are 
stored in sequence in the frame memory 38 through a bus 44. 

Figs. 3, 4 and 5 respectively show a plurality of scene change detection algorithms. Fig. 3 shows a method 
55 in which the scene change is determined when a total sum of difference values of image data of the same 
coordinates of two time-continuing frame images, which is obtained for the whole frame, exceeds a certain 
threshold value (T). Fig. 4 shows a method in which the scene change is determined when a value obtained 
by counting the number of pixels for which the difference value of image data of the same coordinates of two 
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time-continuing frame images is larger than the threshold value (T1) exceeds a certain threshold value (T2). 
Fig. 5 shows a method in which the scene change is determined when the shapes of histograms which are 
obtained respectively for two time-continuing frame images are not identical. In this connection, the similarity 
of the shape of histogram can be determined from the total sum of difference values of respective frequency 

5 values in the histograms. 

When the scene change i detected with respect to an input dynamic image at the time of input storage as 
described above, the frame position information of the input dynamic images of the scene start frame and the 
scene end frame is obtained for each scene. Consequently, the number of scenes and a pair of the scene start 
frame number and the scene end frame number are added to the input dynamic image as shown in Fig. 6. 

10 In this embodiment, the characteristics amount extraction means extracts a length of each scene (the num- 
ber of framed between scene changes) as a characteristics amount This amount can be calculated by sub- 
tracting the start frame number from the end frame number of each scene and adding 1 to the result obtained. 
In addition, the relational information of dynamic images is also furnished by the user at the time of input stor- 
age and respective dynamic images are stored in the hard disk device 42 as shown in Fig. 7. In this case, the 

15 dynamic image name, kind, image source and input date are simultaneously stored as an example of relational 
information. 

Adynamic image serving as a search key is entered at the time of retrieval, the above-described scene 
change automatic detection and characteristics amount extraction are carried out for such dynamic image 
which serves as the search key and a sequence of characteristics amounts (number of frames) is obtained. 

20 In this embodiment, when the number of scenes is 3 or less, a message "an input dynamic image which serves 
the key is excessively short" is displayed and the retrieval is not carried out. When the number of scenes of 
input dynamic images is 4 or over, the first and last scenes are ignored and only a sequence of scenes included 
between the first and last scenes is noted. The number of scenes can be other than that of this example. Fig. 
8 shows a process from the search key to extraction of the characteristics amount sequence. 

25 Next, it is determined which characteristics amount sequence stored at the time input storage matches 
with a characteristics amount sequence obtained when the search key is entered. In this embodiment, the de- 
termination algorithm regards the characteristics amount sequence to be obtained from the dynamic image 
simply entered as the search key as a vector, compares it with a characteristics amount stored for the same 
number of sequences as the characteristics amount sequence of the search key, and compares a distance on 

30 a vector space with the threshold value with all partial sequences as candidates. If the distance on the vector 
space is less than the threshold value, it is regarded that there is a possibility of an image to be retrieved, that 
is, an appropriate scene. Fig. 9 shows the determination algorithm. 

When the appropriate scene is obtained, the relational information of the dynamic image retrieved as a 
result of retrieval is obtained and displayed to the user. A dynamic image simultaneously retrieved can be dis- 
ss played for confirmation. 

Though the number of frames between scene changes is used as the characteristics amount in the above 
embodiment, the characteristics amount is not limited to this number of frames and the other characteristics 
amount can be used. In this embodiment, an extremely simple algorithm is used to detect the appropriate scene 
and therefore a faulty result of retrieval may be actually included and such retrieval means rather selection of 

40 a candidate. For improvement of the reliability, the accuracy of retrieval can be raised by storing, for example, 
an image of the center frame of each scene for identifying the candidate for each scene and conducting pattern 
matching with the center frame of each scene similarly obtained from the input image. Pattern matching in 
this case is carried out by, for example, processing the total of difference values between images in reference 
to the threshold value and using the histograms of images. 

45 Though the retrieved image is described with an image of a storage medium such as a hard disk in this 
embodiment, the image to be retrieved is not limited to the above and it is clear that the present invention ap- 
plies to the retrieval of a desired dynamic image from dynamic images included in the database transmitted 
through communications such as personal computer communications. 

As easily known from the above description, the present embodiment enables retrieving with dynamic im- 

50 ages as search keys. In addition, the embodiment enables to retrieve relational information from given dynamic 
images in a short period of time without spending a remarkable duration of time either in input storage or in 
retrieval. 

[Second Embodiment] 

55 

Fig. 10 is a block diagram showing a second embodiment of the present invention. In Fig. 10, reference 
numeral 101 denotes a main storage including software modules such as at least four types of modules de- 
scribed later, software for a multi-window and a work area of a CPU 102. 102 is a CPU (central processing 
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unit) for executing a processing according to the software modules on the main storage 101. 103 is a hard 
disk which stores dynamic images, relational information described later and attribute value information. Data 
is managed by a database management system. 104 is a display which is used for implement an interactive 
function with users including acquirement of various information from users and presentation of information 

5 such as dynamic images to users under the control of the multi-window system such as an X window. 105 is 
an operation unit comprising a mouse and a keyboard for operating this system. 

Fig. 11 is a flow chart illustrating the modules to be stored in the main storage 101 shown in Fig. 10. In 
Fig. 11, 201 is a designated area appearance image pattern management module for obtaining and managing 
the required information through the interactive function with users. In this embodiment, this module simulta- 

10 neously obtains the designated area information and the attribute name information corresponding to the des- 
ignated area which are given in the interactive operation, an appearance image pattern which is required as 
a sample image for detecting the attribute value information from the dynamic image frame to be entered by 
an appearance image pattern determination module 202, and the attribute values for respective appearance 
image pattern and stores the information as management information. 

15 202 is an appearance image pattern determination module, wherein, if the designated area appearance 
image pattern management module determines that the prior designated attribute is provided as a result of 
determination as to whether a designated area in each frame of dynamic images entered according to the pre- 
ceding management information is an attribute designated in the appearance image pattern determination 
module 202, the attribute value information of the corresponding image pattern is outputted to the database 

20 management module 203. 

203 is a database management module which is a functional module, wherein the attribute value informa- 
tion and the dynamic frame sequence which are determined for each dynamic image frame sequence to be 
entered by the appearance image pattern determination module 202 are managed as numerical character in- 
formation and image information in the hard disk 103 and management of these information is implemented 

25 by using general database management systems which are commercially available. Numerical character in- 
formation such as relational information including dynamic image names and dynamic image recording time 
which are entered by the database users in correspondence to the attribute value information outputted from 
the appearance image pattern determination module 202 are managed together with the attribute value infor- 
mation in the form of table in the hard disk 103. An RDB (relational database) which is used as a common da- 

30 tabase management system provides the retrieving function in the framework of the relational operation and 
can therefore be directly used for dynamic image retrieval. 

204 is a dynamic image retrieval/presentation module which presents dynamic images retrieved from the 
database and a certain frame of the dynamic images by using the database management module 203 based 
on the retrieving condition derived from the relational information of the frame name and others from the users 

35 of this retrieving system and the attribute information for more minutely classifying the dynamic images based 
on the relational information. Dynamic image data corresponding to the dynamic image name which represents 
the contents of the dynamic image as a result of retrieval outputted from the database management module 
203 is read out from the hard disk 103 and displayed on the display 104 as the result of retrieval. 

As a practical example in this embodiment, a process is described for obtaining, as the attribute value in- 
40 formation, a strike count information to be displayed on a corner of a TV screen in a baseball TV relay scene 
as the subject dynamic image data of retrieval. 

In an interactive environment using the window system as the base, a programmer of the dynamic image 
database designates, as a designated area, a rectangular area of the TV screen where the strike count is dis- 
played as shown in Fig. 12 with the mouse of the operation unit 105, and further designates the frames which 
45 respectively display the strike count of 0, 1 and 2 as a sample frame of the appearance image pattern while 
scrolling the frames of dynamic images. The attribute name for the designated area and the attribute values 
for respective sample frames are entered from the keyboard of the operation unit 105. 

The following describes the modules to be stored in the main storage 101 shown in Fig. 10. 
The designated area appearance image pattern management module 201 manages the data related to 
so the area designated by the operation unit 105 in the form of table as shown in Fig. 13. Specifically, assuming 
the left-side vertex coordinates of the display as (x, y) = (0, 0), four integral values in total, including the upper 
left vertex coordinates (region-start-x, region-start-y) and the width and height of the square (respectively, a 
region width and a region height), which are the designated area information, and the given attribute name 
("strike_counf) are managed in the form of table. A list of those values which can be simultaneously obtained 
55 is managed by the pointer 205. More specifically, the available attribute values and the sample image data 
(sample pattern) from which the areas of the sample frames corresponding to the attribute values are cut out 
are managed as the arrayed data as shown in Fig. 13. 

The appearance image pattern determination module 202 determines whether or not an attribute value 
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can be set for respective frames of the frame sequence of dynamic images to be entered while referring to 
the table shown in Fig. 13 which is managed by the designated area appearance image pattern management 
module 201. As a result of determination, such attribute value information as an "attribute name = attribute 
value" likewise, for example, "strike_count = 1" is outputted. When it is determined that a corresponding image 
5 pattern has not appeared, such information as "strike_count = nil" is outputted. The attribute value information 
as the above-described output is stored in the hard disk 103 in correspondence to the relational information 
given by the database programmer in the database management module, managed by the database module, 
and used for retrieval. 

Various methods can be assumed as a method for determining the appearance of the image pattern in 
10 the appearance image pattern determination module. In this embodiment a method using template matching 
is described. A pixel value of a sample pattern t for respective attribute values of the designated area is as- 
sumed as Pt (x, y). t refers to one of strike_count_0, strike_counM and strike_count_2, denoting respective 
sample patterns, x and y respectively refer to 0 ^ x ^ region_width-1 and 0^ y ^ region Jieight-1 . The region- 
_width and the regionjieight denote the size of the designated area and indicate the width and height of the 
15 sample pattern as shown in Fig. 13. tf the pixel value of the frame of dynamic images to be entered into the 
appearance image pattern determination module 202 is assumed as Q (x, y), 

SUMt(6, Y) = E |Pt(x, y)- Q(region_start_x + x 
20 + 6 , region_start — y + y + Y ) I 

is calculated in the range of -t ^ 6 and y ^ t (t is a small value of approximately 5, indicating the range of search 
for template matching) and the least values shall be respectively minSUMt If the minimum value of minSUMt 

25 is larger than a certain threshold value in a range where t is available, it means that the result of template match- 
ing is not identical to any of sample patterns and it shall be determined that there is no corresponding attribute 
value ("strike.count = nil"), and otherwise the attribute value corresponding to t which derives the minimum 
value is outputted (for example, "strike_count_ = 1" if t is the minimum as t = strike_count_1). 

The dynamic image retrieval/presentation module 204 reads and displays the relational information of the 

30 dynamic image name indicating the contents of the dynamic image corresponding to the dynamic image of 
the frame which is retrieved as the result of retrieval outputted by the database management module 203 and 
the dynamic image corresponding to the retrieval to indicate the retrieved dynamic image and a certain frame 
in the dynamic image. 

This embodiment has used the method for pattern-matching the sample image data and the designated 
35 area in the input dynamic image frame in retrieval operation. However, in a superposed information such as 
a Telop (television opaque) which is outputted as being synthesized in the dynamic image, the original dynamic 
image portion of the background part is often used as is. An error may be caused in determination of the ap- 
pearance image pattern, depending on the accuracy of template matching by comparing the pixels of the Telop 
part and those of the dynamic image part For designating the area in the appearance image pattern manage- 
40 ment module, those colors to be noted are simultaneously specified. A plurality of colors to be noted can be 
selected by clicking the mouse on the colors to be noted and the accuracy of determination can be improved 
by carrying out the template-matching only for the corresponding areas. 

If the pixel values of both the sample pattern and the input frame are not included in the colors in actual 
pattern matching, the difference values are calculated as 0 and the pixel values of the areas of colors to be 
45 noted are compared. 

In this embodiment, the attribute value information is detected by template-matching the appearance im- 
age pattern of the designated area. However, it is obvious that the present invention is not limited to this method 
and the attribute value information can be detected according to the result of detection of a histogram of the 
designated area. 

so As described above, differing from the related art by which the attribute value information is set from all 
frames of dynamic images, the present invention enables to automatically set the attribute value information 
for respective frames included in the dynamic images according to the image pattern which appears in the 
designated areas of the dynamic images and therefore a lot of time is not required for automatic detection of 
the attribute value information and the attribute color information need not be manually entered, thereby greatly 

55 saving manpower. 

In addition, the attribute information can be stored and managed as numerical and character information 
in general database management modules and therefore the present invention provides an effect that the re- 
trieval related to the contents of dynamic images can be implemented by using flexible retrieving condition 
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description such as relational operation of attribute values. 



Claims 

5 

1. A method of retrieving dynamic images corresponding to dynamic images entered for retrieval from dy- 
namic images of a database, comprising: 

a first extraction step for extracting a characteristics amount in accordance with a scene change 
of a dynamic image of said database; 
10 a second extraction step for extracting a characteristics amount in accordance with a scene change 

from said dynamic image entered for retrieval; and 

a determination step for determining a dynamic image corresponding to said dynamic image en- 
tered for retrieval, which is included in the dynamic images of said database, from said characteristics 
amounts extracted by said first extraction step and said second extraction step. 

15 

2. A method of retrieving dynamic images according to Claim 1 , wherein said characteristics amount in ac- 
cordance with said scene change is the number of frames between scene changes. 

3. A method of retrieving dynamic images according to Claim 1 , wherein said characteristics amount com- 
20 prises a vector composed of a plurality of components and determination in said determination step is 

made in accordance with a distance on a vector space between a characteristics amount extracted by 
said first extraction step and a characteristics amount extracted by said second extraction step. 



4. A method of retrieving dynamic images according to Claim 1 , further comprising: 

a storage step for storing relational information for each characteristics amount extracted by said 
first extraction step. 



5. A method of retrieving dynamic images according to Claim 1 , further comprising: 

a display step for displaying relational information in accordance with said dynamic image entered 
for retrieval according to the determination by said determination step. 

30 

6. A method of retrieving dynamic images according to a aim 1 , wherein said scene change is automatically 
extracted. 



7. A method of retrieving dynamic images according to Claim 5, wherein a dynamic image corresponding 
35 to said dynamic image entered for retrieval is displayed with said relational information according to the 

determination by said determination step. 

8. A method for managing images corresponding to those images entered from images of a database, com- 
prising: 

40 a designating step for designating a specified position in a frame on said image as a specified area; 

an extraction step for extracting an image pattern from said specified area of the image designated 
by said designating step; 

a classification step for classifying said image according to an image pattern extracted by said 
extraction step; and 

45 an addition step for adding the relational information to an image classified by said classification 

step. 

9. A method of managing images according to Claim 8, wherein said specified area is defined by coordinates 
information on the frame. 

50 

10. A method of managing images according to Claim 8, wherein attribute value information is assigned to 
said image pattern. 



11. Amethod of retrieving dynamic images according to Claim 8, wherein said relational information is a name 
indicating the contents of the classified image. 

55 

12. A method of managing images according to Claim 10, further comprising: 

a storage step for storing said classified image attribute information and said relational information 
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10 



in correspondence relationship. 

13. A method of managing images according to Claim 12, further comprising: 
a retrieving step for retrieving an image stored by said storage step according to at least one of 

said attribute value information and said relational information. 

14. A method of managing images according to Claim 1 3, further comprising: 
a reproducing step for reproducing an image retrieved by said retrieving step. 

15. A method of managing images according to Claim 8, wherein said classification step classifies images 
by using pixel values of said image pattern. 

16. A method of managing images according to Claim 8, wherein said classification step classifies images 
by using pixel values of said image pattern and conducting pattern matching. 

*5 17. A method of managing images according to Claim 8, wherein said classification step classifies images 
by using areas of specified colors in said image pattern and conducting pattern matching. 

18. An apparatus for the retrieval of images stored in a database, wherein a user may present a key image 
having a desired retrieval characteristic, and wherein the apparatus includes means for comparing the 

20 retrieval characteristic of the key image with characteristics of images stored in the database. 

19. An apparatus according to claim 18 wherein the database stores motion picture sequences. 



25 



30 



35 



40 



45 



50 



20. An apparatus according to claim 19 wherein characteristic data for motion picture sequences is stored 
with reference to scene changes. 



55 
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FIG. 7 
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